This is a pretty big upgrade and as such I limited it to upgrade of Substrate and those dependencies without which it didn't compile. There are many other dependencies to be updated that I'll take care of in a separate PR.

The first commit upgrades Substrate and changes everything that is broken, so it is a bit larger, further PRs address some deprecations an minor cleanups.

The only deprecation that is not fixed yet is GenericChainSpec::from_genesis, which I now think I know how to fix, but I spent way too much time on it already and will get back to local WIP branch once this PR lands.

Notable upstream changes in Substrate:

• interesting PR for decreasing verbosity of test runtimes (not taken advantage of yet, feel free to send a PR), this is just an example of usage, there was a separate change somewhere that introduced it: migrate babe and authorship to use derive-impl paritytech/polkadot-sdk#1790
• block builder API changes, which would have been internal if we didn't have domains, but conversion was straightforward: sc-block-builder: Remove BlockBuilderProvider paritytech/polkadot-sdk#2099
• chain spec changes:
  • chain-spec: getting ready for native-runtime-free world paritytech/polkadot-sdk#1256 deprecated GenericChainSpec::from_genesis and moved code from frame-system's genesis config to the chain spec. I'm not sure it is a particularly helpful change, but this required to switch from genesis config to chain spec for building genesis state for domain, which is confusing because both have the same method for building storage, so things compiled, but crashed in runtime.
  • sc-chain-spec: add support for custom host functions paritytech/polkadot-sdk#2190 allows host functions in chain spec, which is how WASM runtime can access them in order to build genesis state (genesis state is no longer built with native code, but rather with WASM).
• Build the standard library crates when building the runtimes paritytech/polkadot-sdk#2217 should remove strict requirements about LLVM version since newer Rust versions ship with standard library using features Substrate doesn't like and they solved it by compiling standard library alongside runtime instead, so they control compilation options. Changes to leverage this will be in a separate PR.
• substrate-node: NativeElseWasmExecutor is no longer used paritytech/polkadot-sdk#2521 made it possible to get rid of NativeElseWasmExecutor since native runtime is no longer used by the node. While it wasn't strictly required, I removed NativeElseWasmExecutor usage in a separate commit. It is a bit nicer that service crates now both define and implement host functions, though I'm not 100% happy that runtimes are still using those somewhat implicitly.
• Networking changes:
  • add fallback request for req-response protocols paritytech/polkadot-sdk#2771 is an interesting change for future upgrades, but annoyingly caused annoying breaking API changes
  • Rework the event system of sc-network paritytech/polkadot-sdk#1370 is an even bigger network change that affected all gossip implementations that we have a few of. What I like about it is that now config and notification service are created together and that allowed me to discover that we don't actually use all of the gossip implementations, we created config for domain-client-subnet-gossip, but there was no use for its service. As such this crate was removed in a separate PR.
  • Extract warp sync strategy from ChainSync paritytech/polkadot-sdk#2467 introduced massive sync layer changes in Substrate. Eventually it should be possible for us to inject DSN sync more properly into Substrate architecture and it shouldn't be that far from today.
• feat: compute pallet/storage prefix hash at compile time paritytech/polkadot-sdk#1539 is an underdocumented PR with some questionable API changes, but the good thing is that it should reduce weights for everything by calculating storage prefixes at compile time rather than runtime.

Smaller things in Substrate:

• sp-core: Rename VrfOutput to VrfPreOutput paritytech/polkadot-sdk#2534 renamed VrfOutput to VrfPreOutput and VrfSignature.output to VrfSignature.pre_output
• Improve storage monitor API paritytech/polkadot-sdk#2899 simplified storage API so we can monitor root node directory instead of its database subdirectory
• Implement more useful traits in Slot type paritytech/polkadot-sdk#1595 will allow us to take advantage of better typing around slots without converting to integers first, but will be in a seaprate PR
• Move Cors data structure into reachable place paritytech/polkadot-sdk#2898 allowed us to remove some temporary copy-paste from node CLI
• Ensure correct variant count in Runtime[Hold/Freeze]Reason paritytech/polkadot-sdk#1900 adds more boilerplate to pallet-balances config that from my understanding we don't need to worry about too much

There were some small changes in Frontier, but nothing really notable, feel free to check the diff between subspace-v4 and subspace-v5 in our fork.

A few changes from our fork of Substrate were upstreamed, which is nice, but networking changes caused a bunch of conflicts for our patches, so I had to adapt synced state announcement. I also reconsidered approach for sync pausing that allows us to switch to DSN sync and instead of introducing new sync mode after massive changes in paritytech/polkadot-sdk#2467 that made porting of the previous solution painful and invasive and replaced with atomic boolean instead that strategically disables a single operation. Should have the same effect, and while potentially a bit fragile (they can change sync process to request stuff from more places that this boolean will not help with and everything will continue to compile), the change is limited and hopefully temporary.

Overall here are all our patches on top of upstream right now, one of the biggest and most invasive is paritytech/polkadot-sdk#1598 that upstream consistently ignores 😞

As you can see, there will be a few follow-up PRs with cleanup of dependencies and such, but this is the bulk of the work.

Going forward I think it will be interesting to consider Polkadot-SDK's releases so we don't need to patch Frontier, but some of our changes do unfortunately require modification of Frontier to compile at all, not just upstream changes.

I do encourage you to look through all upstream changes in Substrate and Frontier if you have time to do so to familiarize yourself with what is happening.

Code contributor checklist:

• I have read, understood and followed contributing guide